Proposal for tests architecture. #96
Conversation
There was a problem hiding this comment.
The proposal contains good list of requirements and I propose to include the list to README.md.
Meantime, the proposed redesign isn't clear for me. I see a lot of good proposals which (including the requests/responses generation callbacks) seems can be just integrated into the current framework architecture. Per client/server processes seem really a big change. It makes sense, but I don't see an answer for the most crucial questions about the framework:
- why it takes so much effort to develop a new test even for a developer familiar with the framework
- how and is it possible to rework the framework to make the tests development fast.
Let's discuss the questions in chat. Results of the PR should be added to the README.md and or requirements of #56.
|
|
||
| In both frameworks every test type is implemented as unique test, so it's | ||
| required to write message integrity, fussing, workload and many other test | ||
| cases for the same situation. |
There was a problem hiding this comment.
I still have no idea how to easily make the test framework to split a message into different sets of skbs. Probably we can use TCP_NODELAY and write the message in different parts, but I'm not sure about bufferization on Python side. However, I believe the task is solvable in generic way.
I suppose the the message integrity and probably skb splitting can be done on deproxy side. If we need the same for Nginx tests, then we can just implement the logic as a separate class/module.
All in all, with https://www.youtube.com/watch?v=oO-FMAdjY68 and the requirements above in mind, I believe that the test framework should use OO relatively complex class inheritance for the helping functionality while the tests should just on/off necessary flags for message integrity calculation, sending in different skb and other things. I.e. the helping framework must provide as declarative API as possible and it's not so important how complex it internally is.
|
|
||
| Heavy dependency between tests. When a new feature is added, it's added to the | ||
| test framework core and affects a lot of tests. Sometimes unpredictably and | ||
| badly. |
There was a problem hiding this comment.
Yes, we had them. We need to discuss this in chat considering particular cases. There are different possibilities what happens when we're in the situation - we grow the framework that it can run many different things in future or we just keep case-specific login in the framework. Let's talk on the cases in the chat. The cases should make the framework design problems clear.
| - RPS. - maximum rps. | ||
|
|
||
| - Split. - Boolean value, meaning that requests must be splited in multiple skbs. | ||
| Don't know how to implement it yet. |
There was a problem hiding this comment.
Please have a look on pv(1). TCP_NODELAY also could be an answer.
| from many source interfaces, multiple concurrent connections for each client. | ||
|
|
||
| - Fuzzing tests: Workload tests, but every generated messages are split into | ||
| different number of skbs. |
There was a problem hiding this comment.
For me it seems that the things can be done in deproxy.
|
|
||
| When reduced number of tests is executed, only first group of tests is executed, | ||
| the full set is for first and second groups. All groups are executed in background | ||
| stability tests. |
There was a problem hiding this comment.
Make sense for generic case, but not so much sense if a developer changes low level logic w/o changing any features and he's curious about network operation. I propose to be able to set this in config as well as for particular tests. In this way you can run for example a particular TLS test with message integrity check and/or fuzzing (strictly speaking, skb splitting isn't a real fuzzing) at the same time some tests oriented on the low level logic have sense only with one of few options enabled.
|
|
||
| TLS tests - ? | ||
|
|
||
| Low level TCP testing - ? |
There was a problem hiding this comment.
With the recent TLS changes affecting TCP transmission process we have to test TCP operation (@i-rinat has questions regarding TCP segment sized generated by Tempesta). The question will be even harder for HTTP/3. Probably ScaPy isn't a good option for this, because it'd be too tricky to generate TCP flow by hands. Probably iptables mangling and/or eBPF with usual TCP options like Nagle will suite the best. We already have something in helpers/analyzer.py and helpers/flaky.py.
|
The proposal was discussed and a few issues was created based on it. |
No description provided.